Things I Have Learned ( So Far )

This is an account of what I have learned (so far) about the application of statistics to psychology and the other sociobiomedical sciences. It includes the principles "less is more" (fewer variables, more highly targeted issues, sharp rounding off), "simple is better" (graphic representation, unit weighting for linear composites), and "some things you learn aren't so." I have learned to avoid the many misconceptions that surround Fisherian null hypothesis testing. I have also learned the importance of power analysis and the determination of just how big (rather than how statistically significant) are the effects that we study. Finally, I have learned that there is no royal road to statistical induction, that the informed judgment of the investigator is the crucial element in the interpretation of data, and that things take time. What I have learned (so far) has come from working with students and colleagues, from experience (sometimes bitter) with journal editors and review committees, and from the writings of, among others, Paul Meehl, David Bakan, William Rozeboom, Robyn Dawes, Howard Wainer, Robert Rosenthal, and more recently, Gerd Gigerenzer, Michael Oakes, and Leland Wilkinson. Although they are not always explicitly referenced, many of you will be able to detect their footprints in what follows. Some Things You Learn Aren't So One of the things I learned early on was that some things you learn aren't so. In graduate school, right after World War II, I learned that for doctoral dissertations and most other purposes, when comparing groups, the proper sample size is 30 cases per group. The number 30 seems to have arisen from the understanding that with fewer than 30 cases, you were dealing with "small" samples that required specialized handling with "small-sample statistics" instead of the critical-ratio approach we had been taught. Some of us knew about these exotic small-sample statistics—in fact, one of my fellow doctoral candidates undertook a dissertation, the distinguishing feature of which was a sample of only 20 cases per group, so that he could demonstrate his prowess with small-sample statistics. It wasn't until some years later that I discovered (mind you, not invented) power analysis, one of whose fruits was the revelation that for a two-independent-group-mean comparison with n = 30 per group at the sanctified two-tailed .05 level, the probability that a medium-sized effect would be labeled as significant by the most modern methods (a t test) was only .47. Thus, it was approximately a coin flip whether one would get a significant result, even though, in reality, the effect size was meaningful. My n = 20 friend's power was rather worse (.33), but of course he couldn't know that, and he ended up with nonsignificant results—with which he proceeded to demolish an important branch of psychoanalytic theory.